(19) 



J 



Europaisches Patentamt 
European Patent Office 
Office europeen des brevets 



(12) 



(43) Date of publication: 

19.11.2003 Bulletin 2003/47 

(21) Application number: 03004230.3 

(22) Date of filing: 25.02.2003 



(H) EP 1 363 230 A1 

EUROPEAN PATENT APPLICATION 

(51) mtci7: G06K 9/22, G06F 3/033 



(84) 


Designated Contracting States: 


• Zou, Yu 




AT BE BG CH CY CZ DE DK EE ES Fl FR GB GR 


Haidian District Beijing 100080 (CN) 




HU IE IT LI LU MC NL PT SE SI SK TR 


• Raghupathy, Sashi 




Designated Extension States: 


Redmond, WA 98052 (US) 




AL LT LV MK RO 


• Jones, F. David 






Redmond, WA 98052 (US) 


(30) 


Priority: 14.05.2002 US 143864 


• Lui, Charlton E. 






Redmond, WA 98052 (US) 


(71) 


Applicant: MICROSOFT CORPORATION 


• Wang, Jian 




Redmond, WA 98052 (US) 


Hangzhou Zheijiang Province (CN) 


(72) 


Inventors: 


(74) Representative: Griinecker, Kinkeldey, 


• 


Shilman, Michael M. 


Stockmair & Schwanhausser Anwaltssozietat 




Seattle, WA 98103 (US) 


Maximilianstrasse 58 


• 


Wei, Zile 


80538 Munchen (DE) 




Haidian District Beijing 100080 (CN) 





o 

CO 
CM 

CO 
CD 
CO 



CL 
LU 



(54) Classification analysis of freeform digital ink input 



(57) Flexible and efficient systems and methods for 
analyzing digital or electronic ink may automatically 
classify electronic inkstrokes on a page into one or more 
types of stroke (such as drawing strokes, text strokes, 
music strokes, mathematical strokes, charts, flow- 
charts, tables, graphs, etc.). The systems and methods 
may include an input for receiving input ink data includ- 
ing at least one stroke set and a processor for deter- 
mining the type of stroke contained in the stroke set 




based, at least in part, on information regarding the con- 
textual environment relating to the stroke set. The con- 
textual environment relating to the stroke set may in- 
clude one or more contextual features regarding the 
stroke set. These contextual features may include, for 
example, various features relating to the stroke(s) within 
the first stroke set, features relating to stroke(s) located 
within a predetermined range of the first stroke set, and/ 
orfeatures relating to stroke(s) associated in some man- 
ner with the first stroke set. 
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Description 
TECHNICAL FIELD 

[0001] Aspects of the present invention are directed 
generally to systems, methods, and computer-readable 
media including computer-executable instructions for 
analyzing and classifying handwritten digital ink as con- 
taining one or more different types of ink strokes. 

BACKGROUND 

[0002] Typical computer systems, especially compu- 
tersystems using graphical user interfaces (GUIs), such 
as Microsoft WINDOWS®, are optimized for accepting 
user inputfrom one or more discrete input devices, such 
as a keyboard for entering text, and a pointing device, 
such as a mouse with one or more buttons, for operating 
the user interface. The ubiquitous keyboard and mouse 
interface provides for fast creation and modification of 
documents, spreadsheets, database fields, drawings, 
photos and the like. However, a significant gap exists 
between the flexibility provided by the keyboard and 
mouse interface compared with non-computer (i.e., 
standard) pen and paper. With the standard pen and pa- 
per, a user may edit a document, write in non-horizontal 
directions, write notes in a margin, draw pictures and 
other shapes, link separate sets of notes by connecting 
lines or arrows, and the like. In some instances, a user 
may prefer to use a pen to mark-up a document rather 
than review the document on-screen because of the 
ability to freely make notes outside of the confines of the 
keyboard and mouse interface. 

[0003] Some computer systems, however, permit a 
user to write on a screen (e.g., using a "stylus" or "pen" 
for writing notes on an electronic input screen). For ex- 
ample, the Microsoft READER application permits one 
to add digital ink (also referred to herein as "electronic 
ink" or "ink") to a document. The system stores the ink 
and provides it to a user when requested. Other appli- 
cations (for example, drawing applications as known in 
the art associated with the Palm 3.x and 4.x and Pock- 
etPC operating systems) permit the capture and storage 
of drawings. These drawings may include other proper- 
ties associated with the ink strokes used to make up the 
drawings. For instance, line width and color may be 
stored with the ink. One goal of these systems is to rep- 
licate the look and feel of physical ink being applied to 
a piece of paper. 

[0004] One activity normally reserved for physical ink 
and paper is note taking. Personal notes are unique as 
each user. Some users take notes using complete sen- 
tences, while others jot down thoughts or concepts and 
then link the concepts using arrows and the like. The 
latter type of notes tends to be written at different loca- 
tions on a page and/or at different angles on the page. 
Additionally, some users revisit notes later and add fur- 
ther thoughts, clarify, and/or edit previously recorded 



notes. The value present in handwritten notes may rest 
not only in the actual text of the information recorded, 
but also in the layout of the notes and the juxtaposition 
of some notes with respect to others. Further value may 
5 be added in the speed at which users take notes. 

[0005] The transition from an ink pen and physical pa- 
per note taking arrangement to a computer-based note 
taking arrangement may prove difficult. While computer- 
based note taking systems can provide advantages in- 
fo eluding handwriting recognition functionality, searcha- 
bility, and written text reformatting, users may quickly 
become disoriented when the computer-based system 
does not function as expected. 

[0006] A number of systems for electronically captur- 
es jng, rearranging, and displaying handwriting as digital 
ink are known (for example, the I nkWriter® system from 
Aha! Software, now owned by Microsoft Corporation of 
Redmond, Washington). These systems capture ink 
strokes and group the strokes into characters and 
20 words. Writing in multiple regions on a page, as many 
users do, can quickly result in confusion, for example, if 
information intended to be maintained as separate 
notes is combined by the system into a single, incoher- 
ent note. Also, in some existing systems, drag selection 
25 (akin to holding down a mouse button and dragging to 
select text in a text editor) may select large areas of 
blank space (i.e., white space) on the page. When this 
selected text is cut and pasted (using standard compu- 
ter-based text editing concepts) or otherwise utilized, 
30 the large volume of selected blank space may produce 
an unintended and surprising result. This result is coun- 
terintuitive to the average computer user because con- 
ventional text editing systems work differently. 
[0007] Additionally, some known stylus-based com- 
35 puting systems that capture ink strokes require relatively 
structured ink input in order to function in an acceptable 
manner. For example, users of such systems typically 
are admonished to "write neatly" or "write between the 
lines" in a horizontal orientation or write in a specified 
40 ink input area. Failure to follow these instructions may 
cause recognition errors or other errors when the elec- 
tronic ink is presented to an associated handwriting rec- 
ognition system, thereby limiting the usefulness of the 
system for electronic note taking. Also, some users 
45 quickly become frustrated with these errors and limita- 
tions of the system and/or become frustrated when 
forced to constrain and adapt their handwriting to better 
"work around" the limitations of the system. 
[0008] These shortcomings of existing electronic note 
50 taking systems effectively create barriers to adoption of 
stylus-based computing systems. 

SUMMARY 

55 [0009] The present invention provides flexible and ef- 
ficient systems and methods for analyzing digital or 
electronic ink, as well as computer-readable media for 
performing these methods and operating such systems. 
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More specifically, examples of the present invention re- 
late to systems and methods for automatically classify- 
ing electronic ink strokes on a page into one or more 
types of stroke (such as drawing strokes, text strokes, 
etc.). The systems and methods according to some ex- 
amples of the invention receive input ink data including 
at least one stroke set and determine the type of stroke 
(s) contained in the stroke set based, at least in part, on 
information regarding the contextual environment relat- 
ing to the stroke set. The contextual environment of the 
stroke set may suggest contextual features of the stroke 
set. These contextual features may include, for exam- 
ple, aggregations of various local features of the individ- 
ual stroke(s) contained within the stroke set, various fea- 
tures of the interrelationships between individual 
strokes in the stroke set. and/or various features relating 
to stroke(s) associated in some manner with the stroke 
set. The specific contextual features evaluated and re- 
lied upon may depend on the specific stroke types under 
consideration. 

[0010] These and other features and aspects of the 
invention will be apparent upon consideration of the fol- 
lowing detailed description. 

BRIEF DESCRIPTION OF DRAWINGS 

[001 1] The foregoing summary, as well as the follow- 
ing detailed description, may be better understood when 
read in conjunction with the accompanying drawings, 
which are included by way of example, and not by way 
of limitation with regard to the claimed invention. 
[0012] Figure 1 illustrates a schematic diagram of an 
exemplary general-purpose digital computing environ- 
ment that may be used to implement various aspects of 
the present invention. 

[0013] Figure 2 illustrates an exemplary pen-based 
computing system that may be used in accordance with 
various aspects of the present invention. 
[0014] Figure 3 illustrates an example of an overall 
digital ink processing system that may include classifi- 
cation analysis systems and methods according to this 
invention. 

[0015] Figure 4 illustrates a general example of vari- 
ous procedures or parse engines that may be used to 
provide input data useful in some examples of classifi- 
cation analysis systems and methods according to the 
invention. 

[001 6] Figures 5A and 5B illustrate examples of parse 
trees describing input data used by and output data gen- 
erated by one example of a layout analysis system and 
method useful to provide input data for some examples 
of classification analysis systems and 
[0017] methods according to the invention. 
[0018] Figure 6 illustrates a schematic diagram of an 
example of classification analysis procedures or parse 
engines according to the invention. 
[0019] Figure 7 illustrates a schematic diagram of an- 
other example of classification analysis procedures or 



parse engines according to the invention. 
[0020] Figure 8 illustrates local minima and maxima 
points that assist in defining stroke fragments used in 
some examples of processing steps in the present in- 
5 vention. 

[0021] Figure 9 illustrates a flow diagram for a classi- 
fication analysis procedure useful according to some ex- 
amples of the invention. 

[0022] Figure 1 0 illustrates a schematic diagram of an 
10 example of a system useful in allowing the classification 
analysis procedure or method of the present invention 
to operate at the same time a user is actively entering 
ink into a document. 



[0023] As described above, examples of the present 
invention relate to systems and methods for analyzing 
digital or electronic ink, and particularly for automatically 

20 classifying electronic ink strokes on a page into one or 
more types of stroke (such as drawing type strokes, text 
type strokes, etc.). The following describes various ex- 
amples of the invention in more detail. 
[0024] This specification contains figures that sche- 

25 matically illustrate various methods and systems useful 
in practicing examples of the invention (e.g., Figures 3. 
4, 6, 7, and 10). These schematic illustrations are in- 
tended to generally illustrate examples of both systems 
and/or methods useful in accordance with the invention . 

30 Therefore, in some instances, depending on the context 
of the sentence, a specific element from these figures 
(such as layout analysis element 302, temporal line 
grouping element 408, and the like) may be referred to 
as a system (e.g., a temporal line grouping system 408), 

35 while in other instances that same element and refer- 
ence number may be used in reference to a method, a 
procedure, a step, a parse engine, and/orthe like. All of 
these variations (e.g., systems, methods, steps, proce- 
dures, parse engines, and the like) are intended to be 

40 included within the scope of these figures. 

[0025] The following description is divided into sub- 
sections to assist the reader. The sub-sections include: 
Terms, General-Purpose Computer Classification Anal- 
ysis Overview, Detailed Description of Classification 

45 Analysis, Other Features, and Conclusion. 

I. TERMS 

[0026] The following terms are used in this specifica- 
50 tion: 

[0027] Ink (also called "digital ink" or "electronic ink") 
- A sequence or set of handwritten strokes. A sequence 
of strokes may include strokes in an ordered form. The 
sequence may be ordered by the time the stroke was 
55 captured and/or by where the stroke appears on a page. 
Other orders are possible. 

[0028] Point - Information defining a location in space. 
For example, a point may be defined relative to a cap- 
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turing space (for example, points on a digitizer) and/or 
a display space (the points or pixels of a display device) . 
Points may be represented using a variety of known 
techniques including two dimensional Cartesian coordi- 
nates (X, Y), polar coordinates (r, 0), three dimensional 
coordinates ((X, Y, Z), (r, 0, p), (X, Y, t (where t is time)), 
(r, 0, t)), four dimensional coordinates ((X, Y, Z, t) and 
(r, 0, p, t)), and other techniques as known in the art. 
[0029] Stroke - A sequence or set of captured points. 
A stroke may be determined in a number of ways, for 
example, using time (e.g., a stroke is all points encoun- 
tered by the stylus during a predetermined time interval), 
using a predetermined number of points [e.g., a stroke 
is all points I through X where X is predefined), or using 
stylus contact with the digitizer surface (e.g.. a stroke is 
all points encountered by the stylus between a pen- 
down event and a pen-up event). When rendered, the 
sequence of points may be connected with lines. Alter- 
natively, a stroke may be represented as a point and a 
vector in the direction of the next point. Further, a stroke 
may be referred to as a simple list (or array or table) of 
points. In short, a stroke is intended to encompass any 
representation of points or segments relating to ink, ir- 
respective of the underlying representation of points 
and/or what connects the points. 
[0030] Stroke set - A data set containing information 
regarding a single stroke or a plurality of strokes asso- 
ciated with one another. A stroke set may include a line 
of associated strokes, a block (e.g., a paragraph) of as- 
sociated strokes, or some other association of plural 
strokes. 

[0031] Stroke type -A term describing the general cat- 
egory or characteristic of a stroke or stroke set. Exam- 
ples of stroke types include "drawing type strokes" and 
"writing type strokes." 

[0032] Drawing type strokes - One example of a 
stroke type. Drawing type strokes typically have low lin- 
earity. Examples of drawing type strokes may include: 
free form drawings, flow diagrams, tables, charts, some 
types of mathematics, etc. 

[0033] Writing type strokes - Another example of a 
stroke type. Writing type strokes typically have high lin- 
earity. Examples of writing type strokes may include: 
text, music, some types of mathematics, etc. 
[0034] Contextual environment - With respect to a 
specific stroke or stroke set, the contextual environment 
relates to one or more characteristics of a group of 
strokes that are located within and/or around the spec- 
ified stroke or stroke set. 

[0035] Local features - Features or characteristics of 
a particular stroke. Local features of a stroke may in- 
clude, for example, stroke length, stroke width, stroke 
height, stroke curvature, number of stroke fragments, 
average stroke fragment height or width, median stroke 
fragment height or width, and the like. 
[0036] Contextual features - Features or characteris- 
tics of a group of strokes in some manner associated 
with a specific stroke or stroke set (optionally including 



the characteristics of the specific stroke or stroke set). 
Examples of contextual features of a stroke or stroke 
set include features or characteristics of stroke(s) within 
the same stroke set, features or characteristics of 

5 strokes in proximity to the stroke or stroke set, and/or 
features or characteristics of strokes associated in some 
manner to the stroke or stroke set. More specific exam- 
ples of contextual features include the number of 
strokes or stroke fragments in the stroke set, the number 

10 of strokes or stroke fragments in a line containing the 
stroke set, the number of strokes or stroke fragments in 
a block containing the stroke set, linearity of the stroke 
set, linearity of a line containing the stroke set, linearity 
of lines in a block containing the stroke set, and the like. 

15 [0037] Render - The process of determining how 
graphics (and/or ink) are to be displayed, whether on a 
screen or printed. 

[0038] Parse Tree - A data structure representing the 
structure of a document. Figures 5A and 5B illustrate 
20 examples of parse trees, both before and after a layout 
analysis procedure, wherein a given page of a docu- 
ment is parsed into blocks, lines, words, and individual 
strokes. 

[0039] Parse engine - A single processing step or pro- 
25 cedure in an ink analysis engine. A typical ink analysis 
engine contains several parse engines, each focusing 
on a particular task. One example of an ink analysis en- 
gine is the layout analysis engine described herein, 
which includes individual parse engines for temporal 
30 line grouping, spatial block grouping, spatial line group- 
ing, list detection, and spatial word grouping. A parse 
engine takes a parse tree as input and modifies it (if ap- 
propriate) to produce a parse tree with a different struc- 
ture, which in turn may be passed along as input to the 
35 next parse engine. 

[0040] Stroke fragment - A subsequence of the points 
in a stroke, derived by splitting the stroke at salient 
points, such as points of high curvature (cusps) and/or 
local maxima and minima. 

40 

II. GENERAL-PURPOSE COMPUTER 

[0041] Figure 1 illustrates a schematic diagram of an 
exemplary conventional general-purpose digital com- 

45 puting environment that may be used to implement var- 
ious aspects of the present invention. In Figure 1 . a com- 
puter 100 includes a processing unit 110, a system 
memory 1 20, and a system bus 1 30 that couples various 
system components including the system memory to the 

50 processing unit 110. The system bus 130 may be any 
of several types of bus structures including a memory 
bus or memory controller, a peripheral bus, and a local 
bus using any of a variety of bus architectures. The sys- 
tem memory 1 20 includes read only memory (ROM) 1 40 

55 and random access memory (RAM) 150. 

[0042] A basic input/output system 160 (BIOS), con- 
taining the basic routines that help to transfer informa- 
tion between elements within the computer 100, such 
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as during start-up, is stored in the ROM 140. The com- 
puter 100 also includes a hard disk drive 170 for reading 
from and writing to a hard disk (not shown), a magnetic 
disk drive 1 80 for reading from or writing to a removable 
magnetic disk 190, and an optical disk drive 191 for 
reading from or writing to a removable optical disk 1 92, 
such as a CD ROM or other optical media. The hard disk 
drive 1 70, magnetic disk drive 1 80, and optical disk drive 
191 are connected to the system bus 1 30 by a hard disk 
drive interface 1 92, a magnetic disk drive interface 1 93, 
and an optical disk drive interface 1 94, respectively. The 
drives and their associated computer-readable media 
provide nonvolatile storage of computer readable in- 
structions, data structures, program modules, and other 
data for the personal computer 1 00. It will be appreciat- 
ed by those skilled in the artthat othertypes of computer 
readable media that may store data that is accessible 
by a computer, such as magnetic cassettes, flash mem- 
ory cards, digital video disks, Bernoulli cartridges, ran- 
dom access memories (RAMs), read only memories 
(ROMs), and the like, may also be used in the example 
operating environment. 

[0043] A number of program modules may be stored 
on the hard disk drive 170, magnetic disk 190, optical 
disk 192, ROM 140, or RAM 150, including an operating 
system 1 95, one or more application programs 1 96, oth- 
er program modules 1 97, and program data 1 98. A user 
may enter commands and information into the computer 
1 00 through input devices, such as a keyboard 1 01 and 
a pointing device 102. Other input devices (not shown) 
may include a microphone, joystick, game pad, satellite 
dish, scanner, orthe like. These and other input devices 
often are connected to the processing unit 1 1 0 through 
a serial port interface 1 06 that is coupled to the system 
bus 1 30, but may be connected by other interfaces, such 
as a parallel port, game port, or a universal serial bus 
(USB). Further still, these devices may be coupled di- 
rectly to the system bus 1 30 via an appropriate interface 
(not shown). A monitor 1 07 or other type of display de- 
vice is also connected to the system bus 130 via an in- 
terface, such as a video adapter 108. In addition to the 
monitor 1 07, personal computers typically include other 
peripheral output devices (not shown), such as speak- 
ers and printers. As one example, a pen digitizer 165 
and accompanying pen or user input device 1 66 are pro- 
vided in order to digitally capture freehand input. The 
pen digitizer 1 65 may be coupled to the processing unit 
1 1 0 via the serial port interface 1 06 and the system bus 
1 30, as shown in Figure 1 , orthrough any other suitable 
connection. Furthermore, although the digitizer 165 is 
shown apart from the monitor 1 07, the usable input area 
of the digitizer 1 65 may be co-extensive with the display 
area of the monitor 107. Further still, the digitizer 165 
may be integrated in the monitor 1 07, or may exist as a 
separate device overlaying or otherwise appended to 
the monitor 107. 

[0044] The computer 1 00 may operate in a networked 
environment using logical connections to one or more 



remote computers, such as a remote computer 1 09. The 
remote computer 109 may be a server, a router, a net- 
work PC, a peer device, or other common network node, 
and typically includes many or all of the elements de- 

5 scribed above relative to the computer 1 00, although on- 
ly a memory storage device 1 1 1 with related applications 
programs 1 96 have been illustrated in Figure 1 . The log- 
ical connections depicted in Figure 1 include a local area 
network (LAN) 112 and a wide area network (WAN) 113. 

10 Such networking environments are commonplace in of- 
fices, enterprise-wide computer networks, intranets, 
and the Internet. 

[0045] When used in a LAN networking environment, 
the computer 1 00 is connected to the local network 1 1 2 

15 through a network interface or adapter 1 1 4. When used 
in a WAN networking environment, the personal com- 
puter 100 typically includes a modem 115 or other 
means for establishing a communications link over the 
wide area network 11 3, e.g., to the Internet. The modem 

20 115, which may be intennal or external, is connected to 
the system bus 1 30 via the serial port interface 1 06. In 
a networked environment, program modules depicted 
relative to the personal computer 1 00, or portions there- 
of, may be stored in a remote memory storage device. 

25 [0046] It will be appreciated that the network connec- 
tions shown are exemplary and other techniques for es- 
tablishing a communications link between the comput- 
ers may be used. The existence of any of various well- 
known protocols such as TCP/IP, Ethernet, FTP, HTTP 

30 and the like is presumed, and the system may be oper- 
ated in a client-server configuration to permit a user to 
retrieve web pages from a web-based server. Any of var- 
ious conventional web browsers may be used to display 
and manipulate data on web pages. 

35 [0047] Figure 2 illustrates an exemplary pen-based 
computing system 201 that may be used in accordance 
with various aspects of the present invention. Any or all 
of the features, subsystems, and functions in the system 
of Figure 1 may be included in the computer of Figure 

40 2. Pen-based computing system 201 includes a large 
display surface 202, e.g., a digitizing flat panel display, 
such as a liquid crystal display (LCD) screen, on which 
a plurality of windows 203 is displayed. Using stylus 204, 
a user may select, highlight, and/or write on the digitizing 

45 display surface 202. Examples of suitable digitizing dis- 
play surfaces 202 include electromagnetic pen digitiz- 
ers, such as Mutoh or Wacom pen digitizers. Othertypes 
of pen digitizers, e.g., optical digitizers, may also be 
used. Pen-based computing system 201 interprets ges- 

50 tures made using stylus 204 in order to manipulate data, 
enter text, create drawings, and/or execute convention- 
al computer application tasks, such as spreadsheets, 
word processing programs, and the like. 
[0048] The stylus 204 may be equipped with one or 

55 more buttons or other features to augment its selection 
capabilities. In one example, the stylus 204 may be im- 
plemented as a "pencil" or "pen," in which one end con- 
stitutes a writing element and the other end constitutes 
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an "eraser" end, and which, when moved across the dis- 
play, indicates portions of the display to be erased. Oth- 
er types of input devices, such as a mouse, trackball, or 
the like may be used. Additionally, a user's own finger 
may be the stylus 204 and used for selecting or indicat- 
ing portions of the displayed image on a touch-sensitive 
or proximity-sensitive display. Consequently, the term 
"user input device," as used herein, is intended to have 
a broad definition and encompasses many variations on 
well-known input devices, such as the stylus 204. Re- 
gion 205 shows a feedback region or contact region per- 
mitting the user to determine where the stylus 204 con- 
tacted the display surface 202. 

III. CLASSIFICATION ANALYSIS OVERVIEW 

A. Classification Analysis Systems and Methods 

[0049] The present invention relates to systems and 
methods for analyzing electronic ink input, e.g., in a sty- 
lus-based computing environment. In stylus-based 
computing environments, electronic ink may be intro- 
duced into the system as "strokes" by writing with a sty- 
lus on a digitizing display surface that captures the 
strokes. As one desirable feature provided in at least 
some examples according to this invention , the user has 
free reign to write anywhere on the digitizing display sur- 
face, in any orientation, just like a user would have with 
conventional pen and paper. In these examples of the 
invention, the user's input is not confined to any partic- 
ular computer or line orientation, stroke size, timing, or 
in any other manner. Moreover, the user need not advise 
the system in advance of the type of strokes that he/she 
intends to enter (e.g., no need to preset a drawing mode, 
a text mode, a music mode, a math mode, or the like). 
The classification analysis systems and methods ac- 
cording to this invention evaluate a stroke set and de- 
termine the type of stroke represented in the stroke set. 
This classification information then can be used in send- 
ing the ink input data to other appropriate processing 
systems, which may result in better processing of the 
input data strokes. Moreover, proper classification of the 
input stroke data can help the computing system 
present suitable or targeted editing menus and/or other 
menus and/or other information to the user, thereby 
making the overall pen-based computing systems and 
methods more user-friendly. 

[0050] In general, this invention relates to systems 
and methods for classifying input electronic or digital ink 
strokes (e.g., determining whether the ink strokes con- 
stitute writing or drawing, and/or even more particularly, 
whetherthe input ink strokes constitute handwritten text, 
drawings, musical notes, flowcharts, mathematics, 
graphs, charts, tables, etc.). The systems and methods 
may include an input device or step for receiving input 
ink data including at least a first stroke set (which may 
include one or more strokes), and a processor system 
or step for determining a type of stroke contained in the 



first stroke set based, at least in part, on information re- 
garding a first contextual environment relating to the first 
stroke set. As noted above, the "contextual environ- 
ment" of a stroke set relates to one or more character- 
5 istics of a group of strokes that are located within and/ 
or around the specified stroke or stroke set. 
[0051] The invention also relates to computer-reada- 
ble media containing computer-executable instructions 
for performing the classification analysis methods and 
operating the classification analysis systems. 
[0052] The "contextual environment" of a stroke set 
may include information relating to at least one contex- 
tual feature associated with the stroke set. This contex- 
tual feature may relate to a feature or characteristic of 
at least one member selected from the group consisting 
of: one or more strokes in the stroke set itself, one or 
more strokes located within a predetermined range of 
the stroke set, and/or one or more strokes associated 
with the stroke set. As a more specific example, the con- 
textual environment or contextual features of a stroke 
set may relate to features or characteristics of a block 
of input ink data including the stroke set (e.g., a para- 
graph containing the stroke set) or a line of input ink data 
including the stroke set. 

[0053] One evaluation or determination made in clas- 
sification procedures according to various examples of 
the invention may include a determination as to whether 
a given stroke set contains drawing type strokes or writ- 
ing type strokes. This determination takes advantage of 
characteristic features of the various types of ink input 
being evaluated or considered. For example, a set of 
handwritten text strokes commonly includes a defined 
structure, often both horizontally and vertically, that is 
typically very linear, with several individual strokes 
present on a given line. Moreover, the individual strokes 
in handwritten text typically have a similar size (usually 
relatively short, particularly their height) and a generally 
"loopy" shape. A set of handwritten drawing strokes, on 
the other hand, typically will exhibit a less globally linear 
structure and less "loopiness," and such sets of strokes 
often contain at least some relatively long strokes that 
in some manner surround other strokes. Accordingly, a 
drawing/writing classification analysis procedure may 
take advantage of one or more of these features char- 
acteristic of handwritten text and drawings in determin- 
ing whether a given stroke set contains writing or draw- 
ing type strokes. 

[0054] The classification analysis systems and meth- 
ods according to various examples of the invention, 
however, need not only generally evaluate for text or 
drawing type classification. Several other different types 
of "writing" strokes may be evaluated and recognized 
without departing from the invention, provided a set of 
characteristics or features can be defined for that type 
of stroke. Some of these other types of writing strokes 
also may have relatively linear contextual features like 
handwritten text. For example, handwritten music may 
tend to have a relatively linear structure. Additionally, at 
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least some mathematical writing may tend to be rela- 
tively linear in character {e.g., simple mathematics, al- 
gebra, calculus, etc.). All of these different types of "writ- 
ing" broadly fall within the scope of the term "writing- 
type" strokes or stroke sets, as used in this specification. 
[0055] Similarly, various different types of non-linear 
drawing type strokes also exist. For example, in addition 
to stroke sets containing free form drawings, stroke sets 
containing tables, graphs, charts, flowcharts, and the 
like generally may be considered "drawing-type" strokes 
or stroke sets in accordance with some examples of this 
invention. Additionally, some types of mathematical cal- 
culations may be better characterized as drawing type 
strokes rather than writing type strokes (for example, 
long division, long columns of numbers for addition or 
subtraction, geometry, etc.). 

[0056] Accordingly, in addition to (or as an alternative 
to) determining whether a given stroke set broadly con- 
stitutes a drawing type stroke set or a writing type stroke 
set, systems and methods according to some examples 
of this invention may more particularly classify a given 
stroke set as containing, for example, handwritten text, 
music, mathematics, tables, graphs, charts, flowcharts, 
free form drawings, etc. Examples of contextual fea- 
tures or characteristics of stroke sets that may be con- 
sidered in classifying stroke sets into these particular 
types may include: 

[0057] Text - high number of strokes or stroke frag- 
ments, high linearity per line, high number of "loopy" 
strokes, relatively similar stroke or stroke fragment siz- 
es, relatively small stroke or stroke fragment sizes, rel- 
atively short stroke or stroke fragment spacings, lines 
that are vertically spaced and horizontally overlapping, 
etc. 

[0058] Mathematics - high number of strokes or stroke 
fragments, high linearity per line, high number of "loopy" 
strokes, relatively similar stroke or stroke fragment siz- 
es, relatively small stroke or stroke fragment sizes, rel- 
atively short stroke or stroke fragment spacings, pres- 
ence of mathematical symbols (e.g., +, -, %, =, <.>, %, 
<, x. >, 3, i, sin, cos. tan. etc.), etc. 
[0059] Music - high number of strokes or stroke frag- 
ments, high linearity per line, relatively similar stroke or 
stroke fragment sizes, relatively small stroke or stroke 
fragment size, presence of musical symbols (e.g., ver- 
tical bars, musical notes, etc.), etc. 
[0060] Tables - long linear strokes that intersect and 
enclose text, column and row structure, regular gridded 
spacing, etc. 

[0061] Graphs - two long linear, perpendicularstrokes 
forming the abscissa and ordinate axes, abscissa and 
ordinate labels, a series of short strokes (optionally 
spaced apart) along the abscissa and ordinate axes, 
long strokes between the abscissa and ordinate axes 
lines, etc. 

[0062] Charts - one or more relatively long, non-loopy 
strokes enclosing one or more long straight strokes (e. 
g., a pie chart), etc. 



[0063] Flowcharts - one or more relatively short, non- 
loopy strokes enclosing text (e.g., a box, circle or oval 
enclosing text), straight strokes (or arrows) between text 
enclosing strokes, etc. 

5 [0064] Of course, various other types of contextual 
environment information or contextual features may be 
considered and relied upon in determining the specific 
type of strokes present in a stroke set without departing 
from this invention. Systems and methods according to 

10 some examples of the invention may be used to classify 
any number of different types of strokes, provided the 
appropriate contextual environment information or con- 
textual features are available for the evaluation and 
classification. 

15 [0065] In addition to looking at the contextual environ- 
ment or contextual features relating to a stroke set, the 
systems and methods according to some examples of 
the invention also may look at "local features" of one or 
more strokes contained in the stroke set as part of the 

20 classification analysis. The "local features" may include 
one or more characteristics or attributes of specific indi- 
vidual strokes within a stroke set. 
[0066] Examples of local features or characteristics of 
individual strokes that may be considered in classifying 

25 stroke sets include: 

[0067] Text - stroke length, stroke curvature, etc. 
[0068] Mathematics - stroke length or shape corre- 
sponding to numbers or mathematical symbols, etc. 
[0069] Music -stroke length andshapecorresponding 

30 to musical notes, etc. 

[0070] Tables - long linear stroke, regular gridded 
spacing, etc. 

[0071] Graphs - long perpendicular "axis" strokes, 
etc. 

35 [0072] Charts - long, non-loopy : closed stroke (e.g., a 
pie chart), etc. 

[0073] Flowcharts - short, non-loopy stroke (e.g., a 
box. circle or oval), straight stroke (or arrow), etc. 
[0074] Other features of individual strokes also may 
40 be considered and evaluated without departing from the 
invention. 

[0075] Classification systems and methods according 
to the invention may form a portion of an overall elec- 
tronic ink processing system or method, an example of 
45 which is described in more detail below. 

B. General System 

[0076] Figure 3 is a flow diagram that illustrates an 
50 example of an overall system and method in which clas- 
sification analysis systems and methods according to 
some examples of this invention may be used. In the 
example of Figure 3, incoming or input strokes 300 first 
are subjected to a layout analysis 302, which may com- 
55 bine and parse the individual strokes into associated 
stroke sets, such as words, lines, blocks, and/or other 
groupings 304, and/or may provide other information re- 
lating to the layout of strokes on a page. While any suit- 



8 



13 



EP 1 363 230 A1 



14 



able systems or methods can be used to provide asso- 
ciated stroke sets for classification analysis without de- 
parting from this invention, one suitable and exemplary 
procedure for providing the input data is described in 
more detail below. 

[0077] After layout analysis 302, the data may be in- 
troduced into a variety of different ink analysis engines. 
In the illustrated system of Figure 3, the data is next in- 
troduced to a classification analysis system or method 
306 according to this invention. The classification anal- 
ysis system or engine 306 determines the type(s) of 
strokes included in the specific input data stroke set (e. 
g., whether individual stroke or stroke set represents 
flow diagrams, freeform drawings, text, music, mathe- 
matics, charts, graphs, etc.). 

[0078] Further processing of the input ink may depend 
on the stroke type recognized by the classification anal- 
ysis system or engine 306. For example, for strokes or 
stroke sets that are classified as writing, the classified 
stroke sets may be sent to a handwriting recognition 
system 310 or another appropriate processing system. 
If necessary or desired, prior to introduction into the 
handwriting recognition system 31 0 or other processing 
system, the lines or blocks of ink data may be "normal- 
ized" using a normalization algorithm or system 308, e. 
g., to place the input text in an optimum orientation for 
analysis by the handwriting recognition system 310 or 
other processing system. Conventional normalization 
systems or methods 308 and/or handwriting recognition 
systems or methods 31 0 may be used without departing 
from the present invention. The data output from the 
handwriting recognition system or method 31 0 may con- 
stitute machine-generated text (e.g., lines, words, par- 
agraphs 312, etc.) usable in any conventional manner, 
such as in conventional word processing systems (e.g., 
Microsoft WORD® or the like), e-mail handling systems, 
etc. 

[0079] As another example, if the classification anal- 
ysis engine 306 recognizes the input strokes or stroke 
sets as containing drawing type strokes, the data may 
then be transferred to an annotation recognition system 
or method 314, which can be used to recognize textual 
information in the drawing. Further processing can pro- 
ceed in any conventional manner. For example, if de- 
sired, the drawings may be "cleaned-up," wherein the 
handwritten annotations may be replaced with machine- 
generated text, handwritten drawing lines or shapes (e. 
g., circles, triangles, rectangles, etc.) may be replaced 
with machine-generated elements, and the like. Also, 
the drawings (either the handwritten versions or later 
machine-generated versions) can be introduced into 
any suitable programs or systems without departing 
from this invention. 

[0080] The classification analysis systems and meth- 
ods according to some examples of the invention also 
may recognize other specific writing or drawing types 
without departing from the invention. For example, a 
classification analysis system may recognize input 



stroke sets as containing music, mathematical informa- 
tion, tables, charts, graphs, flow diagrams, etc., without 
departing from the invention. Such stroke sets, if 
present, could be sent to more specialized recognition 
5 systems and/or to other processing applications (e.g., 
to a music synthesizer, or the like). 

C. Input To The Classification Analysis Engine 

10 [0081] The input data for use in a classification anal- 
ysis engine 306 according to examples of the present 
invention can take on any suitable form. For example, 
in one exemplary procedure as illustrated in Figure 3. 
individual strokes of input ink data are combined togeth- 
*5 er and associated into data sets as a result of a succes- 
sion of decisions made by a layout analysis engine 302, 
which groups or associates various individual strokes 
based on an overall ink layout and statistics obtained 
from the input ink. The layout analysis engine 302 may 
20 provide a hierarchical clustering of ink strokes on a 
page, which allows global statistic calculations over the 
cluster(s). The first stroke grouping decisions are con- 
servative, based on local layout relationships when the 
clusters of ink strokes are small (e.g., clusters repre- 
ss senting individual strokes or relatively short combina- 
tions of strokes). Later stroke grouping decisions can be 
more aggressive, due to the more global statistics col- 
lected from larger clusters (e.g., stroke sizes over a 
longer line, relative stroke spacing, line angles, etc.). 
30 Multiple passes through the input ink data may be con- 
ducted to enable increasingly aggressive decision mak- 
ing in determining whether to merge strokes to form 
stroke sets, such as lines and/or blocks of input ink 
strokes. 

35 [0082] Figure 4 generally illustrates steps or parse en- 
gines involved in one example of an ink layout analysis 
system or method 302 useful in providing input data for 
some examples of the classification systems and meth- 
ods according to this invention. Because of the freedom 

40 provided to a user in inputting digital ink into the systems 
and methods according to these examples of the inven- 
tion (e.g., a user is allowed to write anywhere on a page, 
in any orientation, at any time, using any desired stroke 
size), when the layout analysis procedure 302 begins, 

45 there may be no preliminary information from which to 
determine the proper orientation or type of input data (e. 
g., whetherthe incoming input data 400 is textual, draw- 
ing, mathematic, music, flow diagrams, charts, graphs, 
etc. and/or whether the incoming input data is written 

50 horizontally, on an angle, vertically, etc.). Element 402 
in Figure 4 provides a general graphical representation 
of an input data structure 400. This graphical represen- 
tation 402 is illustrated in more detail in the parse tree 
of Figure 5A. In general, when the layout analysis pro- 

55 cedure 302 begins, it treats every stroke S 500 on a giv- 
en page P 508 as a separate word W 502, every word 
W 502 is treated as a separate line L 504, and every line 
L 504 is treated as a separate block B 506. 
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[0083] While this description of the layout analysis en- 
gine 302 uses terms like "word," "line," and "block," 
these terms are used in this specification as a matter of 
convenience to refer to groups of associated strokes or 
stroke sets. At the time the layout analysis step 302 oc- 
curs in this example of the invention, no final determi- 
nation has been made as to whether individual strokes 
or stroke sets constitute writing, drawings, etc. 
[0084] The layout analysis engine 302 according to 
this example of the invention operates greedily, such 
that during each pass (or operation of each parse en- 
gine) stroke or line merger operations occur, but splits 
do not. Moreover, the engine 302 may be operated with 
tests and tolerances such that undesired merger oper- 
ations do not occur. 

[0085] As a result of the operation of layout analysis 
engine 302, the individual strokes S 500 may be com- 
bined into associated words W 502, lines L 504, and 
blocks B 506, where appropriate. Figure 5B illustrates 
a graphical representation 406 of a possible data struc- 
ture for the data output 404 from the layout analysis en- 
gine 302. As illustrated in Figure 5B, the page 508 over- 
all contains the same stroke information, but certain 
strokes S 500 have been combined or associated to- 
gether to form words W 510, and certain words W 510 
have been joined together to form a line L 512. Of 
course, a word may contain any number of strokes, and 
likewise a line may contain any number of words. Also, 
although not illustrated in the particular parse tree ex- 
ample of Figure 5B, two or more lines also may be joined 
together to form a block 514. 

[0086] Figure 4 provides a schematic overview of one 
example of a suitable layout analysis engine 302 useful 
in producing input ink stroke sets for analysis by a clas- 
sification analysis system or method according to the 
invention. In this example, a first step in the layout anal- 
ysis procedure 302 is a temporal line grouping step 408, 
which generally compares features of temporally adja- 
cent strokes and combines them as lines, if appropriate. 
Various factors may betaken into account in determin- 
ing whether temporally adjacent strokes should be 
grouped together, such as stroke size, inter-stroke spac- 
ing, stroke angle, etc. Once this temporal line grouping 
step 408 is completed, the next step in the analysis 302, 
a spatial block grouping step 41 0, compares the tempo- 
ral line groupings and combines lines that are located 
close to one another as spatial blocks (other criteria also 
may be considered). Again, various factors may be tak- 
en into account in determining whether a spatial block 
grouping should be made, such as stroke size, inter- 
stroke spacing, line angle, etc. 

[0087] The temporally grouped lines (from step 408) 
may be further grouped into longer lines (if appropriate), 
optionally taking into consideration their spatial block re- 
lationship or orientation, in a spatial line grouping step 
412. This spatial line grouping step 412 need not con- 
sider the time of one stroke compared to another stroke, 
although factors in addition to the lines' spatial relation- 



ship ororientation may be taken into consideration, such 
as line angle, stroke size, etc. Also, the results of the 
spatial block grouping procedure 410 may be used as 
a factor in determining whether a spatial line grouping 
5 should be made between two existing temporal line 
groupings (e.g.. if both temporal line groupings lie in a 
common spatial blockgrouping, thetemporal line group- 
ings are more likely to be located on a common line, 
provided their spatial relationship and/or orientation in- 
fo dicate that they may lie on a common line). 

[0088] Once the spatial line groupings have been 
completed, the layout analysis procedure 302 according 
to this example of the invention may then group the in- 
dividual strokes in the lines into one or more spatial word 
15 groupings 41 6. depending, for example, on inter-stroke 
spacing, stroke orientation, stroke size, etc. 
[0089] Figure 4 also illustrates an optional parse en- 
gine or step in broken lines that may be performed in 
the layout analysis 302. This optional step is called "list 
20 detection" 414. Often , when users write a list, they tend 
to write a column of numbers or letters, and then fill in 
the list elements. At other times, users will write out the 
content of the list, and then later add a column of num- 
bers, letters, or bullet points. This list detection engine 
25 41 4 detects these special circumstances and combines 
the number, letter, or bullet point strokes with the corre- 
sponding list element. 

[0090] The various steps in this exemplary ink analy- 
sis engine 302 (Figure 4) may be changed in order or 

30 omitted without departing from the invention . For exam- 
ple, if desired, the spatial line grouping step 412 may 
take place before the spatial block grouping step 410. 
[0091] The output data 404 from the layout analysis 
engine 302 can be used in a classification analysis en- 

35 gine 306, as illustrated in Figure 3, and from there the 
classified data may proceed to other appropriate 
processing engines (e.g., annotation recognition 314, 
handwriting recognition 310, etc.). 
[0092] Of course, other suitable engines or proce- 

40 dures for grouping or associating individual strokes into 
stroke sets can be used without departing from this in- 
vention. Also, if desired, prior to processing, the user 
could indicate to the system that certain strokes always 
should be grouped together (e.g.., by drawing a line 

45 around, highlighting, or otherwise selecting input data 
strokes to be associated together). 

IV. DETAILED DESCRIPTION OF CLASSIFICATION 
ANALYSIS 

50 

[0093] This portion of the specification describes ex- 
amples of the classification analysis procedure or en- 
gine 306 according to the invention in more detail. A 
general classification analysis procedure according to 
55 one example of the invention is illustrated in the sche- 
matic diagram of Figure 6. 

[0094] Initially, in processes and methods according 
to this example of the classification analysis, data relat- 
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ing to a set of electronic ink strokes (i.e.. a stroke set) 
is received or input into the classification analysis sys- 
tem (Step S600). The stroke set may contain electronic 
inkdata in many different forms orformats. Forexample, 
the stroke set may contain information relating to any 
number of strokes, provided the strokes in the stroke set 
are associated with one another in some manner. As 
more specific examples, the strokes in the stroke set 
may represent all of or part of a word or line of stroke 
data obtained from an ink layout analysis system 302 
(e.g., a spatial line grouping, a temporal line grouping, 
or a spatial word grouping from the layout analysis en- 
gine 302 described in conjunction with Figures 3 and 4). 
As another example, the strokes in the stroke set may 
represent all of or part of a block of stroke data obtained 
from an ink layout analysis system 302 (e.g., a spatial 
block grouping from the layout analysis engine 302 de- 
scribed in conjunction with Figures 3 and 4). Any suita- 
ble method and/or system for obtaining and sending 
stroke sets (e.g., words, lines, blocks, paragraphs, etc. 
of associated stroke data) to the classification analysis 
engine 306 can be used without departing from the in- 
vention. 

[0095] Once the stroke set has been ascertained and 
sent to the classification analysis system 306, systems 
and methods according to this example of the invention 
attempt to classify and assign a stroke type to the stroke 
(s) contained in the stroke set. For simplicity, the follow- 
ing description primarily describes discerning writing 
type strokes from drawing type strokes. However, as 
discussed above, various different stroke types, both 
within these general headings or categories and/or in 
addition to these general headings or categories, may 
be evaluated and classified without departing from this 
invention. 

[0096] The next step in the procedure requires eval- 
uation of contextual environment information relating to 
the stroke set (Step S602) to determine whether the 
stroke set is in a drawing type environment or a writing 
type environment. Contextual environment relates to 
one or more characteristics of a group of strokes that 
are located within and/or around the given stroke or 
stroke set being evaluated. In Step S604, if the contex- 
tual environment information indicates that the stroke 
set forms a drawing (or part of a drawing), the stroke set 
is classified as drawing type. Alternatively, if the contex- 
tual environment information indicates that the stroke 
forms writing (or part of a writing), the stroke set is clas- 
sified as writing type in Step S604. As described above, 
stroke sets may be classified more specifically and/or in 
other classifications without departing from the inven- 
tion. The procedure then ends (Step S606), or alterna- 
tively, forwards the resulting data from the classification 
analysis procedure 306 to another step or processing 
engine in the overall process (e.g., to a normalization 
system 308, a handwriting recognition system 310, an 
annotation recognition system 314, a music synthesiz- 
ing system, or other suitable processing system). 



[0097] Figure 7 schematically illustrates another ex- 
ample of a system or method according to the invention. 
Again, the procedure starts by receiving data relating to 
a stroke set to be classified (Step S700). Then, one or 

5 more local features of at least one individual stroke with- 
in the stroke set are evaluated (Step S702). While any 
suitable features of a stroke may be evaluated without 
departing from the invention, some examples of the in- 
vention that classify between writing type strokes and 

10 drawing type strokes evaluate the individual stroke 
length and stroke curvature as the local features of a 
stroke in the stroke set. In general, handwritten text con- 
tains a relatively large number of strokes that are rela- 
tively short in length and relatively curvy or loopy. As an 

15 example of this step of the procedure, systems and 
methods according to the invention may look at each 
individual stroke in the stroke set and determine the per- 
centage of strokes in the stroke set that are curvy or 
loopy. Stroke sets that contain a large percentage of 

20 curvy or loopy strokes are more likely to contain hand- 
written text as compared to stroke sets containing a low 
percentage of curvy or loopy strokes. Additionally, 
stroke sets that contain relatively short and/or consist- 
ently sized strokes or stroke fragments also are more 

25 likely to contain handwritten text as compared to stroke 
sets in drawings, which are more likely to contain rela- 
tively long and inconsistently sized strokes or stroke 
fragments. As one example, a stroke set may be re- 
quired to contain 60% or more drawing type strokes (a 

30 g. } long and/or non-loopy strokes) before the stroke set 
may be classified as drawing type. The percentage may 
be changed, if desired, for example, depending on the 
number of strokes in the stroke set and/or the overall 
length of the line containing the stroke set. 

35 [0098] If desired, the local features of the strokes in 
the stroke set may be characterized in several different 
ways. For example, in the procedure of Step S702, the 
system or method may determine the percentage of 
"loopy-long" strokes in the stroke set, the percentage of 

40 "loopy-short" strokes, the percentage of "straight-long" 
strokes, and the percentage of "straight-short" strokes, 
and then usethis information as local features in making 
the classification or determination as to whether the 
stroke set contains writing type strokes or drawing type 

45 strokes. 

[0099] The next step in the exemplary procedure il- 
lustrated in Figure 7 involves evaluation of contextual 
features relating to the stroke set (Step S704). Contex- 
tual features of a stroke set relate to characteristics of 

50 a group of strokes that are in some manner associated 
with a specific stroke or stroke set being classified (op- 
tionally including the characteristics of the specific 
stroke or stroke set being classified). Examples of con- 
textual features of a stroke or stroke set include features 

55 or characteristics of strokes within the same stroke set, 
features or characteristics of strokes in proximity to the 
stroke or stroke set, and/or features or characteristics 
of strokes associated in some manner to the stroke or 
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stroke set. 

[01 00] Some specific examples of contextual features 
relating to a stroke set that may be used in classifying 
or discerning writing stroke sets from drawing stroke 
sets include: the number of strokes or stroke fragments 
in the stroke set, the number of strokes or stroke frag- 
ments in a word or line containing the stroke set, the 
number of strokes or stroke fragments in a block con- 
taining the stroke set, the linearity of the stroke set, the 
linearity of a word or line containing the stroke set, and 
the linearity of lines in a block containing the stroke set. 
[0101] Stroke number in a stroke set may be readily 
determined, e.g., by counting the number of pen-down 
to pen-up events within the stroke set, or in a word, line, 
or block containing the stroke set, etc. However, be- 
cause some writing styles (such as cursive handwritten 
text) may contain relatively long continuous strokes (for 
example, as a person writes a lengthy word) ; it may be 
advantageous in some examples of the invention to uti- 
lize the number of stroke fragments, rather than the 
number of strokes. Figure 8 illustrates an example of a 
series of strokes broken into its corresponding stroke 
fragments. In one example, a "stroke fragment" may be 
considered to be a portion of a stroke obtained by break- 
ing a stroke at its local minima and maxima points, when 
the baseline of the stroke is treated as horizontal. As 
shown in Figure 8, several of the individual strokes in 
the sentence "This is a line" contain plural stroke frag- 
ments. For example, the single stroke "a" (reference 
number 800) as written in this figure contains four dif- 
ferent stroke fragments 802, 804, 806, and 808. Break- 
ing a stroke into fragments tends to "normalize" cursive 
and printed handwriting (i.e., stroke fragments of the 
cursive word "hello" appear relatively similar to stroke 
fragments of the printed word "hello"). Moreover, break- 
ing a long stroke into stroke fragments provides a larger 
sample size when calculating statistics relating to the 
stroke set (e.g., more reference points from which to cal- 
culate average or median stroke fragment height, width, 
etc.). 

[0102] The number of fragments is simply a total 
count of the stroke fragments in the stroke set, or in the 
word, line, or block containing the stroke set, or in 
strokes associated with the stroke set, and the like. Gen- 
erally, text or other writing will contain a relatively large 
number of stroke fragments as compared to drawings 
(which tend to have relatively large numbers of straight 
lines). Therefore, if a stroke set (e.g., a word, line, or 
block) contains a large number of fragments (e.g., 9 or 
more stroke fragments per line), there is a greater like- 
lihood that this stroke set contains handwritten text, and 
the stroke set is considered to contain writing type 
strokes. In this example, stroke sets containing 8 frag- 
ments per line or less are considered to contain drawing 
type strokes. 

[0103] Another contextual feature of a stroke set that 
may provide information as to whether the stroke set 
contains drawing or writing type strokes is the linearity 



of the stroke set itself, the linearity of a word, line, or 
block containing the stroke set, the linearity of other 
strokes associated with the stroke set, and/or the like. 
One example of a way of measuring stroke linearity is 
5 through the use of the stroke set's fragment centroid er- 
ror. The centroid error for a stroke fragment, as used in 
this example, is the distance that the fragment's centroid 
lies from a regression line that best fits a line of strokes 
in or containing the stroke set. The fragment centroid 
error in a line of a stroke set may be considered to be 
the sum of centroid errors for each stroke fragment in 
the line or stroke set. As noted above, handwritten text 
(as well as certain other handwritten writing types) is 
generally quite linear, and the stroke sizes are typically 
relatively short. Accordingly, low centroid error means 
that the stroke fragments are located relatively close to 
the regression line, which means that the stroke frag- 
ments are more linear, and thus more likely to contain 
text. As one specific example, if the ratio of the fragment 
centroid error to line width for a stroke set is 0.2 or less, 
then the stroke set may be considered to contain writing 
type strokes, whereas if this ratio is greater than 0.2, the 
stroke set may be considered to contain drawing type 
strokes. If desired, the fragment centroid error for a 
stroke set may be normalized based on the number of 
stroke fragments in the stroke set (e.g., total fragment 
centroid error/number of stroke fragments), and appro- 
priate threshold values can be determined using this 
normalized fragment centroid error value. 
[0104] Returning to Figure 7, after the local and con- 
textual features of the stroke set have been determined, 
the stroke set type is then determined based on this in- 
formation (Step S706). Any suitable classification algo- 
rithm can be used without departing from the invention. 
For instance, a support vector machine (SVM) with ra- 
dial basis function, a Bayesian classifier, a neural net- 
work, and the like may be used to perform this classifi- 
cation step without departing from the invention. Alter- 
natively, the classification analysis could be based on 
an appropriate decision tree (e.g., a linear decision 
tree) . 

[0105] After the classification step S706 is completed, 
the procedure may end (Step S708), or the resulting in- 
formation may be used in a next step in the overall proc- 
ess, e.g., to send the data or stroke sets to appropriate 
recognition systems or to other suitable processing sys- 
tems or methods. 

[0106] Figure 9 illustrates an exemplary algorithm that 
may be used in classification analysis procedures ac- 
cording to some examples of this invention. In this ex- 
ample, the classification analysis determines whether 
the stroke set contains writing type strokes or drawing 
type strokes. In the procedure, first the stroke set data 
is received or input into the system (Step S900), which 
can occur in any suitable manner, such as from a user 
entering ink into a stylus-based computing system, 
downloading from memory or another source, etc. At 
Step S902, the system determines whether the number 
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of stroke fragments present in the stroke set exceeds a 
predetermined threshold limit (identified as "X" in Figure 
9). If YES, the stroke set is designated as containing 
writing type strokes (Step S904), and the procedure 
ends (Step S906). Optionally, although not specifically 
illustrated in Figure 9, the procedure could begin clas- 
sification analysis of a new stroke set, reanalyze a mod- 
ified version of a previously analyzed stroke set, or oth- 
erwise proceed to another processing step. 
[0107] The threshold level X can be set by the skilled 
artisan in any appropriate manner, depending, for ex- 
ample, on the overall size of the stroke set, the number 
of strokes in the stroke set, and the like. As one specific 
example, X is set at 8, such that stroke sets (e.g., lines 
of stroke data) containing 8 or fewer stroke fragments 
are considered to possibly contain drawing type strokes, 
whereas stroke sets containing 9 or more stroke frag- 
ments are considered to contain writing type strokes. 
[0108] If, in Step S902, it is determined that the 
number of fragments in the stroke set is X or less (an- 
swer NO), the system then counts the number of indi- 
vidual drawing type strokes in the stroke set (Step 
S908). As described above, a determination of whether 
an individual stroke is potentially a drawing type stroke 
may be made, for example, by looking at the individual 
stroke length and stroke curvature. Drawings typically 
contain at least some longer strokes and some strokes 
that are less curvy (e.g., having fewer stroke fragments). 
This step S910 in the procedure counts the number of 
strokes in the stroke set that have the characteristics of 
drawing type strokes. 

[0109] Then, in Step S910, the system determines 
whether the ratio of the number of potential drawing type 
strokes in the stroke set to the overall total number of 
strokes in the stroke set exceeds a threshold level (iden- 
tified as "Y" in Figure 9). If YES. the stroke set is desig- 
nated as containing drawing type strokes (Step S912), 
and the procedure ends (Step S906) or otherwise 
moves forward. 

[01 1 0] The threshold value Y can be set at any appro- 
priate level by the skilled artisan, based on routine ex- 
perimentation. As one specific example, the Y value is 
set at 60%, such that stroke sets that contain 60% or 
more drawing type strokes are designated drawing type 
stroke sets, whereas stroke sets containing less than 
60% drawing type strokes are not automatically classi- 
fied as drawing type stroke sets. 
[0111] If, at Step S910, the answer is NO (i.e., the ra- 
tio of drawing type strokes to total strokes is less than 
the threshold Y), the system then determines whether 
the fragment centroid error for the stroke set indicates 
that the stroke set contains drawing type strokes or writ- 
ing type strokes (i.e., the "linearity" of the stroke set is 
considered). In the illustrated procedure, the system de- 
termines whether the ratio of the stroke set's fragment 
centroid error to the width of the entire stroke set is 
greater than a predetermined threshold value Z (Step 
S91 4). If YES, the stroke set is designated as containing 



drawing type strokes (Step S912), and the procedure 
ends (Step S906) or otherwise moves forward. If NO, 
the stroke set is designated as containing writing type 
strokes (Step S916), and then the procedure ends (Step 

5 S906) or otherwise moves forward. The threshold value 
Z also can be set at any appropriate level. As noted 
above, in one example, stroke sets having a fragment 
centroid error to width ratio of 0.2 or below may be con- 
sidered writing type stroke sets, whereas stroke sets 

10 having a ratio greater than 0.2 may be considered draw- 
ing type strokes. 

[0112] As is readily apparent, the algorithm and pro- 
cedure illustrated in Figure 9 and described above are 
merely exemplary. The various steps, threshold levels, 
15 and order of steps may be readily changed and/or mod- 
ified by the skilled artisan without departing from the in- 
vention. 

V. OTHER FEATURES 

20 

[0113] The classification analysis procedure 306 ac- 
cording to this example of the invention can be applied 
to electronic ink in any suitable manner, for example, as 
a post ink entry process, on a page-by-page basis. Ad- 

25 vantageously, however, the classification analysis pro- 
cedure will operate incrementally, as the user generates 
and adds ink in a stylus-based computing environment. 
[0114] Figure 10 illustrates a general schematic dia- 
gram of a system in which classification analysis may 

30 proceed incrementally, as user 1300 adds ink to a page. 
First, the application in which the user 1300 operates 
will have a document tree data structure 1302. In order 
to make the documenttree data structure 1 302 available 
for processing while the user 1300 adds additional ink 

35 to the tree 1 302, the parser will contain a mirror copy of 
the documenttree data structure 1302. The mirror copy 
is called a "mirror tree" data structure 1 304 in Figure 9, 
and this data structure 1304 changes as changes are 
made to the document tree data structure 1302. Once 

40 the mirror tree 1304 is produced, "snapshots" of the mir- 
ror tree 1 304 at any point in time may be transferred to 
the parser thread 1 306 and/or to a handwriting recogni- 
tion thread 1308. The parser thread 1306 and/or the 
handwriting recognition thread 1308 may operate in the 

45 "background," while the user 1300 potentially adds ad- 
ditional ink to the document tree data structure 1302 in 
the application program. When the parser thread 1306 
and/or handwriting recognition thread 1308 complete 
their operations on the mirror tree snapshot, they send 

50 the results back to the original application, updating the 
document tree data structure 1302, which updates are 
mirrored by the mirror tree data structure 1304. New 
"snapshots" can then be taken (including any new ink 
added by the user since the previous snapshot), and the 

55 parser thread 1306 and/or recognition thread 1308 can 
operate on the new snapshot (optionally focusing on 
changes made since the previous snapshot was ana- 
lyzed). The classification analysis systems according to 
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examples of the invention may operate in the parser, for 
example, as part of the parser thread 1 306. 
[0115] In this manner, the classification analysis sys- 
tems and methods according to these examples of the 
invention can incrementally operate as changes are 
made to the original document 1 302, which can reduce 
processing time, at least from the user's point of view. 

VI. CONCLUSION 

[0116] While the invention has been described in 
terms of various specific examples, these specific ex- 
amples merely exemplify the invention and do not limit 
it. Moreover, the fact that a specific feature or function 
of the invention is described in conjunction with a spe- 
cific example does not mean that this feature or function 
is limited to use with that specific example of the inven- 
tion. Rather, unless otherwise specified, the various fea- 
tures and functions described above may be used freely 
in any example of the invention. Also, while specific ex- 
amples are provided in this specification, those skilled 
in the art will be able to determine appropriate tests and 
set appropriate threshold levels for classifying different 
types of strokes through the use of routine experimen- 
tation. Those skilled in the art also will appreciate that 
changes and modifications may be made to the exem- 
plified versions of the invention without departing from 
the spirit and scope of the invention, as defined in the 
appended claims. 



Claims 

1 . A method for classifying ink strokes, comprising: 

receiving input ink data including at least a first 
stroke set; and 

determining a type of stroke contained in the 
first stroke set based, at least in part, on infor- 
mation regarding a first contextual environment 
relating to the first stroke set. 

2. A method according to claim 1 , wherein the deter- 
mining step determines whether the first stroke set 
contains a drawing type stroke or a writing type 
stroke. 

3. A method according to claim 1, wherein the first 
contextual environment includes information relat- 
ing to a block of input ink data including the first 
stroke set. 

4. A method according to claim 1, wherein the first 
contextual environment includes information relat- 
ingto a line of input ink data including the first stroke 
set. 

5. A method according to claim 4, wherein the first 



contextual environment includes information relat- 
ing to a number of strokes or stroke fragments in 
the line of input ink data including thefirst stroke set. 

5 6. A method according to claim 5, wherein the first 
contextual environment further includes information 
relating to linearity of the line of input ink data in- 
cluding the first stroke set. 

10 7. A method according to claim 4, wherein the first 
contextual environment includes information relat- 
ing to linearity of the line of input ink data including 
the first stroke set. 

15 8. A method according to claim 1, wherein the first 
contextual environment includes information relat- 
ing to one or more individual strokes included in the 
first stroke set or one or more individual strokes that 
lie within a predetermined range of the first stroke 
20 set. 

9. A method according to claim 1 , wherein the deter- 
mining step additionally is based on information re- 
lating to a first characteristic of a first individual 

25 stroke included in the first stroke set. 

10. A method according to claim 9, wherein the first 
characteristic includes information relating to the 
first individual stroke's length. 

30 

1 1 . A method according to claim 10, wherein the deter- 
mining step additionally is based on information re- 
lating to asecond characteristic of thefirst individual 
stroke. 

35 

12. Amethod according to claim 11 , wherein thesecond 
characteristic includes information relating to the 
first stroke's curvature. 

40 13. A method according to claim 9, wherein the first 
characteristic includes information relating to the 
first stroke's curvature. 

14. A method for classifying ink strokes, comprising: 



45 

receiving input ink data including at least a first 
stroke set containing at least a first stroke; and 
determining a type of stroke contained in the 
first stroke set, wherein the determining is 

50 based on: (a) at least one local feature relating 

to one or more individual strokes in the first 
stroke set; and (b) at least one contextual fea- 
ture relating to at least one member selected 
from the group consisting of: one or more 

55 strokes in the first stroke set, one or more 

strokes located within a predetermined range 
of the first stroke set, and one or more strokes 
associated with the first stroke set. 
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15. A method according to claim 14, wherein the deter- 
mining step determines whether the first stroke set 
contains a drawing type stroke or a writing type 
stroke. 

16. A method according to claim 14, wherein the con- 
textual feature includes information relating to a 
block of input ink data including the first stroke set. 

17. A method according to claim 14, wherein the con- 
textual feature includes information relating to a line 
of input ink data including the first stroke set. 

18. A method according to claim 17, wherein the con- 
textual feature includes information relating to a 
number of strokes or stroke fragments in the line of 
input ink data including the first stroke set. 

19. A method according to claim 18, wherein the con- 
textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 

20. A method according to claim 17, wherein the con- 
textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 

21. A method according to claim 14, wherein the local 
feature includes information relating to a first char- 
acteristic of at least one individual stroke included 
in the first stroke set. 

22. A method according to claim 21, wherein the first 
characteristic includes information relating to at 
least one individual stroke's length. 

23. A method according to claim 22, wherein the local 
feature further includes information relating to a 
second characteristic of at least one individual 
stroke. 

24. A method according to claim 23, wherein the sec- 
ond characteristic includes information relating to at 
least one individual stroke's curvature. 

25. A method according to claim 21 , wherein the first 
characteristic includes information relating to at 
least one individual stroke's curvature. 

26. A method for classifying ink strokes, comprising: 

receiving input ink data including at least a first 
stroke set containing at least a first stroke; and 
determining a type of stroke contained in the 
first stroke set, wherein the determining is 
based on at least one contextual feature relat- 
ing to at least one member selected from the 
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20 



group consisting of: one or more strokes in the 
first stroke set, one or more strokes located 
within a predetermined range of the first stroke 
set, and one or more strokes associated with 
the first stroke set. 

27. A method according to claim 26, wherein the deter- 
mining step determines whether the first stroke set 
contains a drawing type stroke or a writing type 
stroke. 

28. A method according to claim 26, wherein the con- 
textual feature includes information relating to a 
block of input ink data including the first stroke set. 

29. A method according to claim 26, wherein the con- 
textual feature includes information relating to a line 
of input ink data including the first stroke set. 

30. A method according to claim 29, wherein the con- 
textual feature includes information relating to a 
number of strokes or stroke fragments in the line of 
input ink data including the first stroke set. 

31. A method according to claim 30, wherein the con- 
textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 

32. A method according to claim 29, wherein the con- 
textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 



35 33. A system for classifying ink strokes, comprising: 



an input device for receiving ink data, wherein 
the ink data includes information relating to at 
least a first stroke set; and 
a processor system for determining a type of 
stroke contained in the first stroke set based, 
at least in part, on information regarding a first 
contextual environment relating to the first 
stroke set. 
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34. A system according to claim 33, wherein the proc- 
essor determines whether the first stroke set con- 
tains a drawing type stroke or a writing type stroke. 

35. A system according to claim 33, wherein the first 
contextual environment includes information relat- 
ing to a block of input ink data including the first 
stroke set. 

36. A system according to claim 33, wherein the first 
contextual environment includes information relat- 
ing to a line of input ink data including the first stroke 
set. 
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37. A system according to claim 36, wherein the first 
contextual environment includes information relat- 
ing to a number of strokes or stroke fragments in 
the line of input ink data including thefirst strokeset. 

38. A system according to claim 37, wherein the first 
contextual environmentfurther includes information 
relating to linearity of the line of input ink data in- 
cluding the first stroke set. 

39. A system according to claim 36, wherein the first 
contextual environment includes information relat- 
ing to linearity of the line of input ink data including 
the first stroke set. 

40. A system according to claim 33, wherein the first 
contextual environment includes information relat- 
ing to one or more individual strokes included in the 
first stroke set or one or more individual strokes that 
lie within a predetermined range of the first stroke 
set. 

41. A system according to claim 33, wherein the proc- 
essor determines the type of stroke based addition- 
ally on information relating to a first characteristic of 
a first individual stroke included in the first stroke 
set. 



group consisting of: at least one stroke in the 
first stroke set, at least one stroke located with- 
in a predetermined range of the first stroke set, 
and at least one stroke associated with the first 
5 stroke set. 

47. A system according to claim 46, wherein the proc- 
essor determines whether the first stroke set con- 
tains a drawing type stroke or a writing type stroke. 

10 

48. A system according to claim 46, wherein the con- 
textual feature includes information relating to a 
block of input ink data including the first stroke set. 

15 49. A system according to claim 46, wherein the con- 
textual feature includes information relating to a line 
of input ink data including the first stroke set. 

50. A system according to claim 49, wherein the con- 
20 textual feature includes information relating to a 

number of strokes or stroke fragments in the line of 
input ink data including the first stroke set. 

51. A system according to claim 50, wherein the con- 
25 textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 



42. A system according to claim 41, wherein the first 
characteristic includes information relating to the 
first individual stroke's length. 

43. A system according to claim 42, wherein the proc- 
essor determines the type of stroke based addition- 
ally on information relating to a second characteris- 
tic of the first individual stroke. 

44. A system according to claim 43, wherein the second 
characteristic includes information relating to the 
first stroke's curvature. 

45. A system according to claim 41, wherein the first 
characteristic includes information relating to the 
first stroke's curvature. 

46. A system for classifying ink strokes, comprising: 



52. A system according to claim 49, wherein the con- 
30 textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 

53. A system according to claim 46, wherein the local 
35 feature includes information relating to a first char- 
acteristic of at least one individual stroke included 
in the first stroke set. 

54. A system according to claim 53, wherein the first 
40 characteristic includes information relating to at 

least one individual stroke's length. 

55. A system according to claim 54, wherein the local 
feature further includes information relating to a 

45 second characteristic of at least one individual 
stroke. 



an input device for receiving ink data, wherein 
the ink data includes information relating to at 
I east a fi rst stro ke set co nta i n i n g at I east a f i rst 50 
stroke; and 

a processor system for determining a type of 
stroke contained in the first stroke set, wherein 
the processor determines the type of stroke 
based on: (a) at least one local feature relating 55 
to at least one individual stroke in thefirst stroke 
set; and (b) at least one contextual feature re- 
lating to at least one member selected from the 



56. A system according to claim 55, wherein the second 
characteristic includes information relating to at 
least one individual stroke's curvature. 

57. A system according to claim 53, wherein the first 
characteristic includes information relating to at 
least one individual stroke's curvature. 

58. A system for classifying ink strokes, comprising: 

an input device for receiving ink data, wherein 
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the ink data includes information relating to at 
least a first stroke set containing at least one 
stroke; and 

a processor system for determining a type of 
stroke contained in the first stroke set, wherein 5 
the processor determines the type of stroke 
based on at least one contextual feature relat- 
ing to at least one member selected from the 
group consisting of: at least one stroke in the 
first stroke set, at least one stroke located with- 10 
in a predetermined range of the first stroke set, 
and at least one stroke associated with the first 
stroke set. 



59. A system according to claim 58, wherein the proc- is 
essor determines whether the first stroke set con- 
tains a drawing type stroke or a writing type stroke. 

60. A system according to claim 58, wherein the con- 
textual feature includes information relating to a 20 
block of input ink data including the first stroke set. 

61. A system according to claim 58, wherein the con- 
textual feature includes information relating to a line 

of input ink data including the first stroke set. 25 

62. A system according to claim 61 , wherein the con- 
textual feature includes information relating to a 
number of strokes or stroke fragments in the line of 
input ink data including the first stroke set. 30 

63. A system according to claim 62, wherein the con- 
textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 35 

64. A system according to claim 63, wherein the con- 
textual feature includes information relating to line- 
arity of the line of input ink data including the first 
stroke set. 40 

65. A computer-readable medium having computer-ex- 
ecutable instructions for performing steps compris- 
ing: 

45 

storing input ink data including at least a first 
stroke set; and 

determining a type of stroke contained in the 
first stroke set based, at least in part, on infor- 
mation regarding a first contextual environment 50 
relating to the first stroke set. 

66. A computer-readable medium having computer-ex- 
ecutable instructions for performing steps compris- 
ing: 55 

storing input ink data including at least a first 
stroke set containing at least a first stroke; and 



determining a type of stroke contained in the 
first stroke set, wherein the determining is 
based on: (a) at least one local feature relating 
to one or more individual strokes in the first 
stroke set; and (b) at least one contextual fea- 
ture relating to at least one member selected 
from the group consisting of: one or more 
strokes in the first stroke set, one or more 
strokes located within a predetermined range 
of the first stroke set, and one or more strokes 
associated with the first stroke set. 

67. A computer-readable medium having computer-ex- 
ecutable instructions for performing steps compris- 
ing: 

storing input ink data including at least a first 
stroke set containing at least a first stroke; and 
determining a type of stroke contained in the 
first stroke set, wherein the determining is 
based on at least one contextual feature relat- 
ing to at least one member selected from the 
group consisting of: one or more strokes in the 
first stroke set, one or more strokes located 
within a predetermined range of the first stroke 
set, and one or more strokes associated with 
the first stroke set. 
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